fix: type hinting fixes and additional code checks #4790
+6,619
−5,223
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
traut
changed the title
traut
added
python
traut
requested review from
Mikaayenson,
eric-forte-elastic and
terrancedejesus
as code owners
Enhancement - GuidelinesThese guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code. Documentation and Context
Code Standards and Practices
Testing
Additional Schema Related Checks
|
traut
changed the title
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Comment on lines
+1418
to
+1395
| osquery_note_pattern = ( | ||
| "> **Note**:\n> This investigation guide uses the [Osquery Markdown Plugin]" | ||
| "(https://www.elastic.co/guide/en/security/current/invest-guide-run-osquery.html) " | ||
| "introduced in Elastic Stack version 8.5.0. Older Elastic Stack versions will display " | ||
| "unrendered Markdown in this guide." | ||
| ) | ||
| invest_note_pattern = ( | ||
| '> This investigation guide uses the [Investigate Markdown Plugin]' | ||
| '(https://www.elastic.co/guide/en/security/current/interactive-investigation-guides.html)' | ||
| ' introduced in Elastic Stack version 8.8.0. Older Elastic Stack versions will display ' | ||
| 'unrendered Markdown in this guide.') | ||
| "> This investigation guide uses the [Investigate Markdown Plugin]" | ||
| "(https://www.elastic.co/guide/en/security/current/interactive-investigation-guides.html)" | ||
| " introduced in Elastic Stack version 8.8.0. Older Elastic Stack versions will display " | ||
| "unrendered Markdown in this guide." | ||
| ) |
There was a problem hiding this comment.
We should double check that when the transform occurs, its still formatted correctly.
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Comment on lines
+250
to
+251
| suggested_path: Path = Path(DEFAULT_PREBUILT_RULES_DIRS[0]) / contents["name"] | ||
| path = Path(path or input(f"File path for rule [{suggested_path}]: ") or suggested_path).resolve() |
There was a problem hiding this comment.
Seems a bit odd to type hint as Path when we explicitly set as a Path object. We also dont type hint the next field path.
There was a problem hiding this comment.
contents is dict[str, Any], so there is ambiguity in the calculation of the result type
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
Mikaayenson
reviewed
Jun 17, 2025
| """Get schema for KQL.""" | ||
| indexes = indexes or () | ||
| converted = flatten_multi_fields(get_schema(version, name='ecs_flat')) | ||
| indexes = indexes or [] |
There was a problem hiding this comment.
Curious as to why this was a tuple
There was a problem hiding this comment.
no idea. I don't think there is a risk of mutation, so we might as well simplify and have list here
|
Co-authored-by: Mika Ayenson, PhD <[email protected]>
|
A recent make test showed so many formatting errors, is this expected @traut ./env/detection-rules-build/bin/python -m ruff format --check
67 files already formatted
./env/detection-rules-build/bin/python -m pyright
/Users/shashankks/elastic_workspace/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/nodeenv.py:26: DeprecationWarning: 'pipes' is deprecated and slated for removal in Python 3.13
import pipes
/Users/shashankks/elastic_workspace/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/nodeenv.py:48: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources import parse_version
* Install prebuilt node (24.2.0) ../Users/shashankks/elastic_workspace/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/nodeenv.py:627: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.
archive.extractall(src_dir, extract_list)
... done.
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:373:5 - error: Type of "client" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:373:14 - error: Type of "authenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:374:5 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:374:12 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:405:9 - error: Type of "pr" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:405:14 - error: Type of "get_pull" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:407:18 - error: Argument type is unknown
Argument corresponds to parameter "repo" in function "check_version_lock_double_bumps" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:407:102 - error: Type of "head" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:407:102 - error: Type of "ref" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:407:102 - error: Argument type is unknown
Argument corresponds to parameter "branch" in function "check_version_lock_double_bumps" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:411:61 - error: Argument type is unknown
Argument corresponds to parameter "repo" in function "check_version_lock_double_bumps" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:418:17 - error: Type of "_" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:418:21 - error: Type of "create_issue_comment" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:425:13 - error: Type of "_" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:425:17 - error: Type of "create_issue_comment" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:652:5 - error: Type of "client" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:652:14 - error: Type of "authenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:653:5 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:653:12 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:783:5 - error: Type of "pr" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:783:10 - error: Type of "create_pull" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:793:9 - error: Type of "add_to_labels" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:796:9 - error: Type of "add_to_assignees" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:799:16 - error: Type of "html_url" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:799:16 - error: Argument type is unknown
Argument corresponds to parameter "message" in function "echo" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:802:50 - error: Type of "html_url" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1592:5 - error: Type of "client" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1592:14 - error: Type of "authenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1601:47 - error: Argument type is unknown
Argument corresponds to parameter "github_client" in function "__init__" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1618:9 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1618:16 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1619:9 - error: Type of "contents" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/devtools.py:1619:20 - error: Type of "get_contents" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:16:20 - error: "Github" is unknown import symbol (reportAttributeAccessIssue)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:16:20 - error: Type of "Github" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:25:24 - error: Type of parameter "github_client" is unknown (reportUnknownParameterType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:26:9 - error: Type of "repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:26:21 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:34:9 - error: Type of "main_branch" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:34:23 - error: Type of "repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:34:23 - error: Type of "get_branch" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:35:9 - error: Type of "main_branch_sha" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:35:27 - error: Type of "commit" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:35:27 - error: Type of "sha" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:37:9 - error: Type of "contents" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:37:20 - error: Type of "repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/endgame.py:37:20 - error: Type of "get_contents" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:26:20 - error: "Github" is unknown import symbol (reportAttributeAccessIssue)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:26:20 - error: Type of "Github" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:133:9 - error: Type of "client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:134:9 - error: Type of "unauthenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:144:9 - error: Return type is unknown (reportUnknownParameterType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:147:16 - error: Type of "__authenticated_client" is partially unknown
Type of "__authenticated_client" is "Unknown | None" (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:148:13 - error: Type of "__authenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:149:16 - error: Type of "__authenticated_client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:149:16 - error: Return type is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:205:33 - error: Type of "client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:205:33 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:206:34 - error: Type of "repo" is partially unknown
Type of "repo" is "Unknown | Repository" (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:206:34 - error: Argument type is partially unknown
Argument corresponds to parameter "repo" in function "get_gh_release"
Argument type is "Unknown | Repository" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:285:9 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:285:16 - error: Type of "client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:285:16 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:286:34 - error: Argument type is unknown
Argument corresponds to parameter "repo" in function "get_gh_release" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:303:9 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:303:16 - error: Type of "client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:303:16 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:307:13 - error: Type of "release" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:307:24 - error: Type of "get_releases" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:308:13 - error: Type of "name" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:308:20 - error: Type of "tag_name" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:309:13 - error: Type of "asset" is partially unknown
Type of "asset" is "Unknown | None" (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:309:26 - error: Argument type is partially unknown
Argument corresponds to parameter "i" in function "next"
Argument type is "Generator[Unknown, None, None]" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:309:33 - error: Type of "a" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:309:38 - error: Type of "get_assets" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:309:62 - error: Type of "name" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:311:29 - error: Argument type is unknown
Argument corresponds to parameter "element" in function "add" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:313:57 - error: Type of "browser_download_url" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/ghwrap.py:313:57 - error: Argument type is unknown
Argument corresponds to parameter "url" in function "load_json_gh_asset" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:648:5 - error: Type of "repo" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:648:12 - error: Type of "client" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:648:12 - error: Type of "get_repo" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:5 - error: Type of "open_prs" is partially unknown
Type of "open_prs" is "list[Unknown]" (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:23 - error: Type of "r" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:28 - error: Type of "get_pulls" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:78 - error: Type of "name" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:78 - error: Argument type is partially unknown
Argument corresponds to parameter "iterable" in function "__init__"
Argument type is "Generator[Unknown, None, None]" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:91 - error: Type of "lbl" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:650:98 - error: Type of "get_labels" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:680:9 - error: Type of "pr" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:13 - error: Argument type is partially unknown
Argument corresponds to parameter "iterable" in function "extend"
Argument type is "list[tuple[Unknown, Unknown]]" (reportUnknownArgumentType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:26 - error: Type of "f" is unknown (reportUnknownVariableType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:31 - error: Type of "get_files" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:49 - error: Type of "filename" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:49 - error: Type of "startswith" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:85 - error: Type of "filename" is unknown (reportUnknownMemberType)
/Users/shashankks/elastic_workspace/detection-rules/detection_rules/rule_loader.py:682:85 - error: Type of "endswith" is unknown (reportUnknownMemberType)
97 errors, 0 warnings, 0 informations
make: *** [lint] Error 1 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Issue link(s):
Summary - What I changed
ruffandpyrightchecks in CI workflowpyrighthas no complainsHow To Test
Checklist
bug,enhancement,schema,maintenance,Rule: New,Rule: Deprecation,Rule: Tuning,Hunt: New, orHunt: Tuningso guidelines can be generatedmeta:rapid-mergelabel if planning to merge within 24 hoursContributor checklist